Skip to contentSkip to conversation

mm: add a new option MREMAP_DUP to mmrep syscall

RFC PATCH In review [email protected] 8 participants 18 messages May 9, 2013 – Jan 6, 2014

Subsystems (7)
Memory management VFS KVM MM core +3 more subsystems
lore
Share & tools
lore
#0 lore raw

cover mm: add a new option MREMAP_DUP to mmrep syscall

From: Wenchao Xia <[email protected]>

This serial try to enable mremap syscall to cow some private memory region,
just like what fork() did. As a result, user space application would got a mirror of those region, and it can be used as a snapshot for further processing.

This patch is based on the commit a12183c62717ac4579319189a00f5883a18dff08 pulled from upstream (linux 3.9) on 2013-04-04, but I hope to sent it first to see if some case I missed to handle correctly, will try rebase to latest upstream code in next version.

simple test code:

#define _GNU_SOURCE
#include <sys/mman.h>
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>

int main(void)

{
    int len = 4096 * 2 ;
    void *old_addr = malloc(len);
    old_addr = ((unsigned long)(old_addr + 4096) & (~0x3FF));
    printf("mapping addr %p with len %d.\n", old_addr, len);
    char *oldc = old_addr;
    oldc[0] = 0;
    oldc[1] = 1;
    oldc[2] = 2;
    oldc[3] = 3;
    void *new_addr;
    unsigned long new_addr_l;
    new_addr = mremap(old_addr, len, 0, 4);
    printf("result new addr %lx %p.\n", new_addr_l, new_addr);
    char *newc = new_addr;
    printf("old value is 0x%lx.\n", *((unsigned long *)oldc));
    printf("new value is 0x%lx.\n", *((unsigned long *)newc));
    newc[0] = 6;
    printf("old value is 0x%lx.\n", *((unsigned long *)oldc));
    printf("new value is 0x%lx.\n", *((unsigned long *)newc));
    oldc[0] = 9;
    printf("old value is 0x%lx.\n", *((unsigned long *)oldc));
    printf("new value is 0x%lx.\n", *((unsigned long *)newc));
    assert(0 == munmap(new_addr, len));

}

Wenchao Xia (6):
mm: add parameter remove_old in move_huge_pmd()
mm : allow copy between different addresses for copy_one_pte() mm : export rss vec helper functions
mm : export is_cow_mapping()
mm : add parameter remove_old in move_page_tables
mm : add new option MREMAP_DUP to mremap() syscall

 fs/exec.c                 |    2 +-
 include/linux/huge_mm.h   |    2 +-
 include/linux/mm.h        |    9 ++-
 include/uapi/linux/mman.h |    1 +
 mm/huge_memory.c          |    6 +-
 mm/memory.c               |   33 ++++----
 mm/mremap.c               |  200 +++++++++++++++++++++++++++++++++++++++++++--
 7 files changed, 224 insertions(+), 29 deletions(-)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#1 lore raw

RFC PATCH 1/6 mm: add parameter remove_old in move_huge_pmd()

From: Wenchao Xia <[email protected]>

---
 include/linux/huge_mm.h |    2 +-
 mm/huge_memory.c        |    6 ++++--
 mm/mremap.c             |    2 +-
 3 files changed, 6 insertions(+), 4 deletions(-)
include/linux/huge_mm.h +1 -1
index ee1c244..567dc1e 100644
@@ -29,7 +29,7 @@ extern int move_huge_pmd(struct vm_area_struct *vma,
 			 struct vm_area_struct *new_vma,
 			 unsigned long old_addr,
 			 unsigned long new_addr, unsigned long old_end,
-			 pmd_t *old_pmd, pmd_t *new_pmd);
+			 pmd_t *old_pmd, pmd_t *new_pmd, bool remove_old);
 extern int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
 			unsigned long addr, pgprot_t newprot,
 			int prot_numa);
mm/huge_memory.c +4 -2
index e2f7f5a..f752388 100644
@@ -1402,10 +1402,11 @@ int mincore_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
 	return ret;
 }
 
+/* This function copy or moves pmd in same mm */
 int move_huge_pmd(struct vm_area_struct *vma, struct vm_area_struct *new_vma,
 		  unsigned long old_addr,
 		  unsigned long new_addr, unsigned long old_end,
-		  pmd_t *old_pmd, pmd_t *new_pmd)
+		  pmd_t *old_pmd, pmd_t *new_pmd, bool remove_old)
 {
 	int ret = 0;
 	pmd_t pmd;
@@ -1429,7 +1430,8 @@ int move_huge_pmd(struct vm_area_struct *vma, struct vm_area_struct *new_vma,
 
 	ret = __pmd_trans_huge_lock(old_pmd, vma);
 	if (ret == 1) {
-		pmd = pmdp_get_and_clear(mm, old_addr, old_pmd);
+		pmd = remove_old ?
+			pmdp_get_and_clear(mm, old_addr, old_pmd) : *old_pmd;
 		VM_BUG_ON(!pmd_none(*new_pmd));
 		set_pmd_at(mm, new_addr, new_pmd, pmd);
 		spin_unlock(&mm->page_table_lock);
mm/mremap.c +1 -1
index 463a257..0f3c5be 100644
@@ -178,7 +178,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
 			if (extent == HPAGE_PMD_SIZE)
 				err = move_huge_pmd(vma, new_vma, old_addr,
 						    new_addr, old_end,
-						    old_pmd, new_pmd);
+						    old_pmd, new_pmd, true);
 			if (err > 0) {
 				need_flush = true;
 				continue;
Signature · 7 lines
1.7.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#2 lore raw

RFC PATCH 2/6 mm : allow copy between different addresses for copy_one_pte()

In reply to [email protected]

From: Wenchao Xia <[email protected]>

This function now can be used in pte copy in same process with different addresses. It is also exported.

---
 include/linux/mm.h |    4 ++++
 mm/memory.c        |   27 ++++++++++++++-------------
 2 files changed, 18 insertions(+), 13 deletions(-)
include/linux/mm.h +4 -0
index 7acc9dc..68f52bc 100644
@@ -963,6 +963,10 @@ int walk_page_range(unsigned long addr, unsigned long end,
 		struct mm_walk *walk);
 void free_pgd_range(struct mmu_gather *tlb, unsigned long addr,
 		unsigned long end, unsigned long floor, unsigned long ceiling);
+unsigned long copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
+			   pte_t *dst_pte, pte_t *src_pte,
+			   unsigned long dst_addr, unsigned long src_addr,
+			   struct vm_area_struct *vma, int *rss);
 int copy_page_range(struct mm_struct *dst, struct mm_struct *src,
 			struct vm_area_struct *vma);
 void unmap_mapping_range(struct address_space *mapping,
mm/memory.c +14 -13
index 494526a..0357cf1 100644
@@ -824,15 +824,15 @@ out:
 }
 
 /*
- * copy one vm_area from one task to the other. Assumes the page tables
- * already present in the new task to be cleared in the whole range
- * covered by this vma.
+ * copy one pte from @src_addr to @dst_addr. Assumes the page tables and vma
+ * already present in the @dst_addr, @src_addr and @src_pte is covered by
+ * @vma, @rss is a array of size NR_MM_COUNTERS used by caller to sync. dst_mm
+ * may be equal to src_mm. Return 0 or swap entry value.
  */
-
-static inline unsigned long
-copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
-		pte_t *dst_pte, pte_t *src_pte, struct vm_area_struct *vma,
-		unsigned long addr, int *rss)
+unsigned long copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
+			   pte_t *dst_pte, pte_t *src_pte,
+			   unsigned long dst_addr, unsigned long src_addr,
+			   struct vm_area_struct *vma, int *rss)
 {
 	unsigned long vm_flags = vma->vm_flags;
 	pte_t pte = *src_pte;
@@ -872,7 +872,8 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 					 */
 					make_migration_entry_read(&entry);
 					pte = swp_entry_to_pte(entry);
-					set_pte_at(src_mm, addr, src_pte, pte);
+					set_pte_at(src_mm, src_addr,
+						   src_pte, pte);
 				}
 			}
 		}
@@ -884,7 +885,7 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 	 * in the parent and the child
 	 */
 	if (is_cow_mapping(vm_flags)) {
-		ptep_set_wrprotect(src_mm, addr, src_pte);
+		ptep_set_wrprotect(src_mm, src_addr, src_pte);
 		pte = pte_wrprotect(pte);
 	}
 
@@ -896,7 +897,7 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 		pte = pte_mkclean(pte);
 	pte = pte_mkold(pte);
 
-	page = vm_normal_page(vma, addr, pte);
+	page = vm_normal_page(vma, src_addr, pte);
 	if (page) {
 		get_page(page);
 		page_dup_rmap(page);
@@ -907,7 +908,7 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 	}
 
 out_set_pte:
-	set_pte_at(dst_mm, addr, dst_pte, pte);
+	set_pte_at(dst_mm, dst_addr, dst_pte, pte);
 	return 0;
 }
 
@@ -951,7 +952,7 @@ again:
 			continue;
 		}
 		entry.val = copy_one_pte(dst_mm, src_mm, dst_pte, src_pte,
-							vma, addr, rss);
+						addr, addr, vma, rss);
 		if (entry.val)
 			break;
 		progress += 8;
Signature · 7 lines
1.7.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#3 lore raw

RFC PATCH 3/6 mm : export rss vec helper functions

In reply to [email protected]

From: Wenchao Xia <[email protected]>

---
 include/linux/mm.h |    2 ++
 mm/memory.c        |    4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)
include/linux/mm.h +2 -0
index 68f52bc..5071a44 100644
@@ -963,6 +963,8 @@ int walk_page_range(unsigned long addr, unsigned long end,
 		struct mm_walk *walk);
 void free_pgd_range(struct mmu_gather *tlb, unsigned long addr,
 		unsigned long end, unsigned long floor, unsigned long ceiling);
+void init_rss_vec(int *rss);
+void add_mm_rss_vec(struct mm_struct *mm, int *rss);
 unsigned long copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 			   pte_t *dst_pte, pte_t *src_pte,
 			   unsigned long dst_addr, unsigned long src_addr,
mm/memory.c +2 -2
index 0357cf1..add1562 100644
@@ -643,12 +643,12 @@ int __pte_alloc_kernel(pmd_t *pmd, unsigned long address)
 	return 0;
 }
 
-static inline void init_rss_vec(int *rss)
+void init_rss_vec(int *rss)
 {
 	memset(rss, 0, sizeof(int) * NR_MM_COUNTERS);
 }
 
-static inline void add_mm_rss_vec(struct mm_struct *mm, int *rss)
+void add_mm_rss_vec(struct mm_struct *mm, int *rss)
 {
 	int i;
 
Signature · 7 lines
1.7.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#4 lore raw

RFC PATCH 4/6 mm : export is_cow_mapping()

In reply to [email protected]

From: Wenchao Xia <[email protected]>

---
 include/linux/mm.h |    1 +
 mm/memory.c        |    2 +-
 2 files changed, 2 insertions(+), 1 deletions(-)
include/linux/mm.h +1 -0
index 5071a44..9bd01f5 100644
@@ -965,6 +965,7 @@ void free_pgd_range(struct mmu_gather *tlb, unsigned long addr,
 		unsigned long end, unsigned long floor, unsigned long ceiling);
 void init_rss_vec(int *rss);
 void add_mm_rss_vec(struct mm_struct *mm, int *rss);
+bool is_cow_mapping(vm_flags_t flags);
 unsigned long copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 			   pte_t *dst_pte, pte_t *src_pte,
 			   unsigned long dst_addr, unsigned long src_addr,
mm/memory.c +1 -1
index add1562..e5456e1 100644
@@ -723,7 +723,7 @@ static void print_bad_pte(struct vm_area_struct *vma, unsigned long addr,
 	add_taint(TAINT_BAD_PAGE, LOCKDEP_NOW_UNRELIABLE);
 }
 
-static inline bool is_cow_mapping(vm_flags_t flags)
+bool is_cow_mapping(vm_flags_t flags)
 {
 	return (flags & (VM_SHARED | VM_MAYWRITE)) == VM_MAYWRITE;
 }
Signature · 7 lines
1.7.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#5 lore raw

RFC PATCH 5/6 mm : add parameter remove_old in move_page_tables

In reply to [email protected]

From: Wenchao Xia <[email protected]>

---
 fs/exec.c          |    2 +-
 include/linux/mm.h |    2 +-
 mm/mremap.c        |   97 ++++++++++++++++++++++++++++++++++++++++++++++++---
 3 files changed, 93 insertions(+), 8 deletions(-)
fs/exec.c +1 -1
index a96a488..12721e1 100644
@@ -603,7 +603,7 @@ static int shift_arg_pages(struct vm_area_struct *vma, unsigned long shift)
 	 * process cleanup to remove whatever mess we made.
 	 */
 	if (length != move_page_tables(vma, old_start,
-				       vma, new_start, length, false))
+				       vma, new_start, length, false, true))
 		return -ENOMEM;
 
 	lru_add_drain();
include/linux/mm.h +1 -1
index 9bd01f5..a5eb34c 100644
@@ -1085,7 +1085,7 @@ vm_is_stack(struct task_struct *task, struct vm_area_struct *vma, int in_group);
 extern unsigned long move_page_tables(struct vm_area_struct *vma,
 		unsigned long old_addr, struct vm_area_struct *new_vma,
 		unsigned long new_addr, unsigned long len,
-		bool need_rmap_locks);
+		bool need_rmap_locks, bool remove_old);
 extern unsigned long do_mremap(unsigned long addr,
 			       unsigned long old_len, unsigned long new_len,
 			       unsigned long flags, unsigned long new_addr);
mm/mremap.c +91 -6

138 lines not on this page · raw message

Signature · 7 lines
1.7.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#6 lore raw

RFC PATCH 6/6 mm : add new option MREMAP_DUP to mremap() syscall

In reply to [email protected]

From: Wenchao Xia <[email protected]>

This option allow user space program getting a mirror for mem, that is two virtual mapping. The content is COW so it can be used for snapshot a region of mem.

Now shared memory is not COWED yet.

---
 include/uapi/linux/mman.h |    1 +
 mm/mremap.c               |  103 ++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 102 insertions(+), 2 deletions(-)
include/uapi/linux/mman.h +1 -0
index ade4acd..5cf7816 100644
@@ -5,6 +5,7 @@
 
 #define MREMAP_MAYMOVE	1
 #define MREMAP_FIXED	2
+#define MREMAP_DUP	4
 
 #define OVERCOMMIT_GUESS		0
 #define OVERCOMMIT_ALWAYS		1
mm/mremap.c +101 -2

133 lines not on this page · raw message

Signature · 7 lines
1.7.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#7 lore raw
In reply to [email protected]
Quoted 6 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:

From: Wenchao Xia <[email protected]>

This serial try to enable mremap syscall to cow some private memory region,
just like what fork() did. As a result, user space application would got a mirror of those region, and it can be used as a snapshot for further processing.

What not just fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.

Signature · 8 lines
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#8 lore raw

ao? 2013-5-9 22:13, Mel Gorman a??e??:

Quoted 8 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:
Quoted 6 lines

From: Wenchao Xia <[email protected]>

This serial try to enable mremap syscall to cow some private memory region,
just like what fork() did. As a result, user space application would got a mirror of those region, and it can be used as a snapshot for further processing.

Quoted 5 lines

What not just fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.

It have some troubles: parent - child communication, sometimes
page copy.
I'd like to snapshot qemu guest's RAM, currently solution is:
1) fork()
2) pipe guest RAM data from child to parent.
3) parent write down the contents.

To avoid complex communication for data control, and file content
protecting, So let parent instead of child handling the data with a pipe, but this brings additional copy(). I think an explicit API cow mapping an memory region inside one process, could avoid it, and faster and cow less pages, also make user space code nicer.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#9 lore raw
Quoted 26 lines · wenchao
Quoted 12 lines · Mel Gorman
Quoted 10 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:

From: Wenchao Xia <[email protected]>

This serial try to enable mremap syscall to cow some private memory
region,
just like what fork() did. As a result, user space application would got a
mirror of those region, and it can be used as a snapshot for further processing.

Quoted 5 lines

What not just fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.

It have some troubles: parent - child communication, sometimes
page copy.
I'd like to snapshot qemu guest's RAM, currently solution is:
1) fork()
2) pipe guest RAM data from child to parent.
3) parent write down the contents.

Quoted 5 lines

To avoid complex communication for data control, and file content
protecting, So let parent instead of child handling the data with a pipe, but this brings additional copy(). I think an explicit API cow mapping an memory region inside one process, could avoid it, and faster and cow less pages, also make user space code nicer.

A new Linux-specific API is not portable and not available on existing hosts. Since QEMU supports non-Linux host operating systems the fork() approach is preferable.

If you're worried about the memory copy - which should be benchmarked
- then vmsplice(2) can be used in the child process and splice(2) can be used in the parent. It probably doesn't help though since QEMU scans RAM pages to find all-zero pages before sending them over the socket, and at that point the memory copy might not make much difference.

Perhaps other applications can use this new flag better, but for QEMU I think fork()'s portability is more important than the convenience of accessing the CoW pages in the same process.

Stefan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#10 lore raw

CCed KVM guys.

Quoted 44 lines · Stefan Hajnoczi
Quoted 26 lines · wenchao
Quoted 12 lines · Mel Gorman
Quoted 10 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:

From: Wenchao Xia <[email protected]>

This serial try to enable mremap syscall to cow some private memory
region,
just like what fork() did. As a result, user space application would got a
mirror of those region, and it can be used as a snapshot for further processing.

Quoted 5 lines

What not just fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.

It have some troubles: parent - child communication, sometimes
page copy.
I'd like to snapshot qemu guest's RAM, currently solution is:
1) fork()
2) pipe guest RAM data from child to parent.
3) parent write down the contents.

Quoted 5 lines

To avoid complex communication for data control, and file content
protecting, So let parent instead of child handling the data with a pipe, but this brings additional copy(). I think an explicit API cow mapping an memory region inside one process, could avoid it, and faster and cow less pages, also make user space code nicer.

A new Linux-specific API is not portable and not available on existing hosts. Since QEMU supports non-Linux host operating systems the fork() approach is preferable.

If you're worried about the memory copy - which should be benchmarked
- then vmsplice(2) can be used in the child process and splice(2) can be used in the parent. It probably doesn't help though since QEMU scans RAM pages to find all-zero pages before sending them over the socket, and at that point the memory copy might not make much difference.

Quoted 3 lines

Perhaps other applications can use this new flag better, but for QEMU I think fork()'s portability is more important than the convenience of accessing the CoW pages in the same process.

Yup, I agree with you that the new syscall sometimes is not a good solution.

Currently, we're working on live-update[1] that will be enabled on Qemu firstly, this feature let the guest run on the new Qemu binary smoothly without restart, it's good for us to do security-update.

In this case, we need to move the guest memory on old qemu instance to the new one, fork() can not help because we need to exec() a new instance, after that all memory mapping will be destroyed.

We tried to enable SPLICE_F_MOVE[2] for vmsplice() to move the memory without memory-copy but the performance isn't so good as we expected: it's due to some limitations: the page-size, lock, message-size limitation on pipe, etc. Of course, we will continue to improve this, but wenchao's patch seems a new direction for us.

To coordinate with your fork() approach, maybe we can introduce a new flag for VMA, something like: VM_KEEP_ONEXEC, to tell exec() to do not destroy this VMA. How about this or you guy have new idea? Really appreciate for your suggestion.

[1] http://marc.info/?l=qemu-devel&m=138597598700844&w=2
[2] https://lkml.org/lkml/2013/10/25/285

#11 lore raw
Quoted 68 lines · Xiao Guangrong

CCed KVM guys.

Quoted 44 lines · Stefan Hajnoczi
Quoted 26 lines · wenchao
Quoted 12 lines · Mel Gorman
Quoted 10 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:

From: Wenchao Xia <[email protected]>

This serial try to enable mremap syscall to cow some private memory
region,
just like what fork() did. As a result, user space application would got a
mirror of those region, and it can be used as a snapshot for further processing.

Quoted 5 lines

What not just fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.

It have some troubles: parent - child communication, sometimes
page copy.
I'd like to snapshot qemu guest's RAM, currently solution is:
1) fork()
2) pipe guest RAM data from child to parent.
3) parent write down the contents.

Quoted 5 lines

To avoid complex communication for data control, and file content
protecting, So let parent instead of child handling the data with a pipe, but this brings additional copy(). I think an explicit API cow mapping an memory region inside one process, could avoid it, and faster and cow less pages, also make user space code nicer.

A new Linux-specific API is not portable and not available on existing hosts. Since QEMU supports non-Linux host operating systems the fork() approach is preferable.

If you're worried about the memory copy - which should be benchmarked
- then vmsplice(2) can be used in the child process and splice(2) can be used in the parent. It probably doesn't help though since QEMU scans RAM pages to find all-zero pages before sending them over the socket, and at that point the memory copy might not make much difference.

Quoted 3 lines

Perhaps other applications can use this new flag better, but for QEMU I think fork()'s portability is more important than the convenience of accessing the CoW pages in the same process.

Yup, I agree with you that the new syscall sometimes is not a good solution.

Currently, we're working on live-update[1] that will be enabled on Qemu firstly, this feature let the guest run on the new Qemu binary smoothly without restart, it's good for us to do security-update.

In this case, we need to move the guest memory on old qemu instance to the new one, fork() can not help because we need to exec() a new instance, after that all memory mapping will be destroyed.

We tried to enable SPLICE_F_MOVE[2] for vmsplice() to move the memory without memory-copy but the performance isn't so good as we expected: it's due to some limitations: the page-size, lock, message-size limitation on pipe, etc. Of course, we will continue to improve this, but wenchao's patch seems a new direction for us.

Quoted 7 lines

To coordinate with your fork() approach, maybe we can introduce a new flag for VMA, something like: VM_KEEP_ONEXEC, to tell exec() to do not destroy this VMA. How about this or you guy have new idea? Really appreciate for your suggestion.

[1] http://marc.info/?l=qemu-devel&m=138597598700844&w=2
[2] https://lkml.org/lkml/2013/10/25/285

Hi,

What is the purpose of snapshotting guest RAM here, in the context of local migration?

#12 lore raw
Quoted 74 lines · Marcelo Tosatti
Quoted 73 lines · Xiao Guangrong

CCed KVM guys.

Quoted 44 lines · Stefan Hajnoczi
Quoted 26 lines · wenchao
Quoted 12 lines · Mel Gorman
Quoted 10 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:

From: Wenchao Xia <[email protected]>

This serial try to enable mremap syscall to cow some private memory
region,
just like what fork() did. As a result, user space application would got a
mirror of those region, and it can be used as a snapshot for further processing.

Quoted 5 lines

What not just fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.

It have some troubles: parent - child communication, sometimes page copy.
I'd like to snapshot qemu guest's RAM, currently solution is:
1) fork()
2) pipe guest RAM data from child to parent.
3) parent write down the contents.

Quoted 5 lines

To avoid complex communication for data control, and file content protecting, So let parent instead of child handling the data with a pipe, but this brings additional copy(). I think an explicit API cow mapping an memory region inside one process, could avoid it, and faster and cow less pages, also make user space code nicer.

A new Linux-specific API is not portable and not available on existing hosts. Since QEMU supports non-Linux host operating systems the fork() approach is preferable.

If you're worried about the memory copy - which should be benchmarked
- then vmsplice(2) can be used in the child process and splice(2) can be used in the parent. It probably doesn't help though since QEMU scans RAM pages to find all-zero pages before sending them over the socket, and at that point the memory copy might not make much difference.

Quoted 3 lines

Perhaps other applications can use this new flag better, but for QEMU I think fork()'s portability is more important than the convenience of accessing the CoW pages in the same process.

Yup, I agree with you that the new syscall sometimes is not a good solution.

Currently, we're working on live-update[1] that will be enabled on Qemu firstly, this feature let the guest run on the new Qemu binary smoothly without restart, it's good for us to do security-update.

In this case, we need to move the guest memory on old qemu instance to the new one, fork() can not help because we need to exec() a new instance, after that all memory mapping will be destroyed.

We tried to enable SPLICE_F_MOVE[2] for vmsplice() to move the memory without memory-copy but the performance isn't so good as we expected: it's due to some limitations: the page-size, lock, message-size limitation on pipe, etc. Of course, we will continue to improve this, but wenchao's patch seems a new direction for us.

To coordinate with your fork() approach, maybe we can introduce a new flag for VMA, something like: VM_KEEP_ONEXEC, to tell exec() to do not destroy this VMA. How about this or you guy have new idea? Really appreciate for your suggestion.

Quoted 5 lines
Quoted 2 lines

[1] http://marc.info/?l=qemu-devel&m=138597598700844&w=2
[2] https://lkml.org/lkml/2013/10/25/285

Hi,

Hi Marcelo,

Quoted 2 lines

What is the purpose of snapshotting guest RAM here, in the context of local migration?

RAM-shapshotting and local-migration are on the different ways. Why i asked for your guy’s suggestion here is beacuse i thought they need do a same thing that moves memory from one process to another in a efficient way. Your idea? :)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#13 lore raw
Quoted 86 lines · Xiao Guangrong
Quoted 74 lines · Marcelo Tosatti
Quoted 73 lines · Xiao Guangrong

CCed KVM guys.

Quoted 44 lines · Stefan Hajnoczi
Quoted 26 lines · wenchao
Quoted 12 lines · Mel Gorman
Quoted 10 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:

From: Wenchao Xia <[email protected]>

This serial try to enable mremap syscall to cow some private memory
region,
just like what fork() did. As a result, user space application would got a
mirror of those region, and it can be used as a snapshot for further processing.

Quoted 5 lines

What not just fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.

It have some troubles: parent - child communication, sometimes page copy.
I'd like to snapshot qemu guest's RAM, currently solution is:
1) fork()
2) pipe guest RAM data from child to parent.
3) parent write down the contents.

Quoted 5 lines

To avoid complex communication for data control, and file content protecting, So let parent instead of child handling the data with a pipe, but this brings additional copy(). I think an explicit API cow mapping an memory region inside one process, could avoid it, and faster and cow less pages, also make user space code nicer.

A new Linux-specific API is not portable and not available on existing hosts. Since QEMU supports non-Linux host operating systems the fork() approach is preferable.

If you're worried about the memory copy - which should be benchmarked
- then vmsplice(2) can be used in the child process and splice(2) can be used in the parent. It probably doesn't help though since QEMU scans RAM pages to find all-zero pages before sending them over the socket, and at that point the memory copy might not make much difference.

Quoted 3 lines

Perhaps other applications can use this new flag better, but for QEMU I think fork()'s portability is more important than the convenience of accessing the CoW pages in the same process.

Yup, I agree with you that the new syscall sometimes is not a good solution.

Currently, we're working on live-update[1] that will be enabled on Qemu firstly, this feature let the guest run on the new Qemu binary smoothly without restart, it's good for us to do security-update.

In this case, we need to move the guest memory on old qemu instance to the new one, fork() can not help because we need to exec() a new instance, after that all memory mapping will be destroyed.

We tried to enable SPLICE_F_MOVE[2] for vmsplice() to move the memory without memory-copy but the performance isn't so good as we expected: it's due to some limitations: the page-size, lock, message-size limitation on pipe, etc. Of course, we will continue to improve this, but wenchao's patch seems a new direction for us.

To coordinate with your fork() approach, maybe we can introduce a new flag for VMA, something like: VM_KEEP_ONEXEC, to tell exec() to do not destroy this VMA. How about this or you guy have new idea? Really appreciate for your suggestion.

Quoted 5 lines
Quoted 2 lines

[1] http://marc.info/?l=qemu-devel&m=138597598700844&w=2
[2] https://lkml.org/lkml/2013/10/25/285

Hi,

Hi Marcelo,

Quoted 7 lines
Quoted 2 lines

What is the purpose of snapshotting guest RAM here, in the context of local migration?

RAM-shapshotting and local-migration are on the different ways. Why i asked for your guy’s suggestion here is beacuse i thought they need do a same thing that moves memory from one process to another in a efficient way. Your idea? :)

Another possibility is to use memory that is not anonymous for guest RAM, such as hugetlbfs or tmpfs.

IIRC ksm and thp have limitations wrt tmpfs.

Still curious about RAM snapshotting.

#14 lore raw
Quoted 95 lines · Marcelo Tosatti
Quoted 86 lines · Xiao Guangrong
Quoted 74 lines · Marcelo Tosatti
Quoted 73 lines · Xiao Guangrong

CCed KVM guys.

Quoted 44 lines · Stefan Hajnoczi
Quoted 26 lines · wenchao
Quoted 12 lines · Mel Gorman
Quoted 10 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:

From: Wenchao Xia <[email protected]>

This serial try to enable mremap syscall to cow some private memory
region,
just like what fork() did. As a result, user space application would got a
mirror of those region, and it can be used as a snapshot for further processing.

Quoted 5 lines

What not just fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.

It have some troubles: parent - child communication, sometimes page copy.
I'd like to snapshot qemu guest's RAM, currently solution is:
1) fork()
2) pipe guest RAM data from child to parent.
3) parent write down the contents.

Quoted 5 lines

To avoid complex communication for data control, and file content protecting, So let parent instead of child handling the data with a pipe, but this brings additional copy(). I think an explicit API cow mapping an memory region inside one process, could avoid it, and faster and cow less pages, also make user space code nicer.

A new Linux-specific API is not portable and not available on existing hosts. Since QEMU supports non-Linux host operating systems the fork() approach is preferable.

If you're worried about the memory copy - which should be benchmarked
- then vmsplice(2) can be used in the child process and splice(2) can be used in the parent. It probably doesn't help though since QEMU scans RAM pages to find all-zero pages before sending them over the socket, and at that point the memory copy might not make much difference.

Quoted 3 lines

Perhaps other applications can use this new flag better, but for QEMU I think fork()'s portability is more important than the convenience of accessing the CoW pages in the same process.

Yup, I agree with you that the new syscall sometimes is not a good solution.

Currently, we're working on live-update[1] that will be enabled on Qemu firstly, this feature let the guest run on the new Qemu binary smoothly without restart, it's good for us to do security-update.

In this case, we need to move the guest memory on old qemu instance to the new one, fork() can not help because we need to exec() a new instance, after that all memory mapping will be destroyed.

We tried to enable SPLICE_F_MOVE[2] for vmsplice() to move the memory without memory-copy but the performance isn't so good as we expected: it's due to some limitations: the page-size, lock, message-size limitation on pipe, etc. Of course, we will continue to improve this, but wenchao's patch seems a new direction for us.

To coordinate with your fork() approach, maybe we can introduce a new flag for VMA, something like: VM_KEEP_ONEXEC, to tell exec() to do not destroy this VMA. How about this or you guy have new idea? Really appreciate for your suggestion.

Quoted 5 lines
Quoted 2 lines

[1] http://marc.info/?l=qemu-devel&m=138597598700844&w=2
[2] https://lkml.org/lkml/2013/10/25/285

Hi,

Hi Marcelo,

Quoted 7 lines
Quoted 2 lines

What is the purpose of snapshotting guest RAM here, in the context of local migration?

RAM-shapshotting and local-migration are on the different ways. Why i asked for your guy’s suggestion here is beacuse i thought they need do a same thing that moves memory from one process to another in a efficient way. Your idea? :)

Quoted 4 lines

Another possibility is to use memory that is not anonymous for guest RAM, such as hugetlbfs or tmpfs.

IIRC ksm and thp have limitations wrt tmpfs.

Yes, KSM and THP are what we're concerning about.

Quoted 2 lines

Still curious about RAM snapshotting.

Wen Chao, could you please tell it more?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#15 lore raw
In reply to wenchao
Quoted 14 lines · wenchao

ao? 2013-5-9 22:13, Mel Gorman a??e??:

Quoted 8 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:
Quoted 6 lines

From: Wenchao Xia <[email protected]>

This serial try to enable mremap syscall to cow some private memory region,
just like what fork() did. As a result, user space application would got a mirror of those region, and it can be used as a snapshot for further processing.

Quoted 5 lines

What not just fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.

Quoted 6 lines

It have some troubles: parent - child communication, sometimes
page copy.
I'd like to snapshot qemu guest's RAM, currently solution is:
1) fork()
2) pipe guest RAM data from child to parent.
3) parent write down the contents.

CC Pavel

I wounder if you can reuse the CRIU approach for memory snapshoting.

http://thread.gmane.org/gmane.linux.kernel/1483158/

Signature · 7 lines
 Kirill A. Shutemov

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#16 lore raw
Quoted 15 lines · Kirill A. Shutemov
Quoted 14 lines · wenchao

ao? 2013-5-9 22:13, Mel Gorman a??e??:

Quoted 13 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:
Quoted 6 lines

From: Wenchao Xia <[email protected]>

This serial try to enable mremap syscall to cow some private memory region,
just like what fork() did. As a result, user space application would got a mirror of those region, and it can be used as a snapshot for further processing.

What not just fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.

Quoted 8 lines
Quoted 6 lines

It have some troubles: parent - child communication, sometimes
page copy.
I'd like to snapshot qemu guest's RAM, currently solution is:
1) fork()
2) pipe guest RAM data from child to parent.
3) parent write down the contents.

CC Pavel

Thank you!

Quoted 1 line

I wounder if you can reuse the CRIU approach for memory snapshoting.

I doubt it. First of all, we need to have task's memory in existing external process which is not its child. With MREMAP_DUP we can't have this. And the most important thing is that we don't need pages duplication on modification. It's the waste of memory for our case. We just need to know the fact that the page has changed.

Wenchao, why can't you use existing KVM dirty-tracking for making mem snapshot? As per my understanding of how KVM MMU works you can

1 turn dirty track on
2 read pages from their original places
3 pick dirty bitmap and read changed pages several times 4 freeze guest
5 repeat step 3
6 release guest

Does it work for you?

This is very very similar to how we do mem snapshot with CRIU (dirty tracking is the soft-dirty patches from the link Kirill provided).

Quoted 2 lines

http://thread.gmane.org/gmane.linux.kernel/1483158/

Thanks,
Pavel

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#17 lore raw

ao? 2013-5-11 22:16, Pavel Emelyanov a??e??:

Quoted 23 lines · Kirill A. Shutemov
Quoted 22 lines
Quoted 20 lines · wenchao

D'D?D? 2013-5-9 22:13, Mel Gorman DuD?D(C)D1D?DGBP:

Quoted 8 lines · On Thu, May 09, 2013 at 05:50:05PM +0800, [email protected] wrote:
Quoted 6 lines

From: Wenchao Xia <[email protected]>

This serial try to enable mremap syscall to cow some private memory region,
just like what fork() did. As a result, user space application would got a mirror of those region, and it can be used as a snapshot for further processing.

Quoted 5 lines

What not just fork()? Even if the application was threaded it should be managable to handle fork just for processing the private memory region in question. I'm having trouble figuring out what sort of application would require an interface like this.

It have some troubles: parent - child communication, sometimes
page copy.
I'd like to snapshot qemu guest's RAM, currently solution is:
1) fork()
2) pipe guest RAM data from child to parent.
3) parent write down the contents.

Quoted 4 lines
Quoted 1 line

CC Pavel

Thank you!

Sorry I forgot CC you, I have viewed the contents on CRIU website
before patching. :>

Quoted 17 lines
Quoted 1 line

I wounder if you can reuse the CRIU approach for memory snapshoting.

I doubt it. First of all, we need to have task's memory in existing external process which is not its child. With MREMAP_DUP we can't have this. And the most important thing is that we don't need pages duplication on modification. It's the waste of memory for our case. We just need to know the fact that the page has changed.

Wenchao, why can't you use existing KVM dirty-tracking for making mem snapshot? As per my understanding of how KVM MMU works you can

1 turn dirty track on
2 read pages from their original places
3 pick dirty bitmap and read changed pages several times 4 freeze guest
5 repeat step 3
6 release guest

Quoted 5 lines

Does it work for you?

This is very very similar to how we do mem snapshot with CRIU (dirty tracking is the soft-dirty patches from the link Kirill provided).

It is different. Actually this approach is already used in qemu as
migration, also as a work around for snapshot. Dirty tracking actually
made a mirror of latest memory region, but for snapshot we need not
sync up the two mirror, and syncing up require frequently changed pages being saved several times. This brings extra trouble for following actions, and also require transfer speed > memory changing speed.
Looking at the block layer's function, for example, LVM2. Most of them
provided an API to do a snapshot by COW, so the idea comes: why not add
an same one for memory? Then the APIs are full for user. Later in
discuss Fork() seems one way to do it, with disadvantage that additional
process comes, so I hope to improve it.
Comparation:
Dirty tracking VS cow

CPU       higher              minimum
Memory    less                higher
I/O       higher              minimum

Since dirty tracking keeps sync with latest memory data, so it is
more ideal for migration than snapshot, but in principle migration is a bit different with snapshot. Further thinking, dirty tracking could work together with cow, to form an incremental snapshot chain, reducing the pages need to be written, but not related to this patch.

Base->delta->delta

Quoted 7 lines
Quoted 2 lines

http://thread.gmane.org/gmane.linux.kernel/1483158/

Thanks,
Pavel

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>